Group Normalization
نویسندگان
چکیده
Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems — BN’s error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN’s usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN’s computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform or compete with its BN-based counterparts for object detection and segmentation in COCO, and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.
منابع مشابه
Normalization and Reliability Evaluation of Persian Version of Two-Pair Dichotic Digits in 8 to 12-Year-Old Children
Objectives: All subjects suspected of Central Auditory Processing Disorder (CAPD) were previously tested by free recall dichotic digits test (DDT). The study objective was normalization and reliability evaluation of two-pair DDT in 750 native Persian subjects aged 8 to 12 years. Materials: A total of 750 subjects were divided into five age groups varying between 8 years and 12 years and 11 mon...
متن کاملThe effect serum vitamin D normalization in preventing recurrences of benign paroxysmal positional vertigo .a case-control study
Background: Benign paroxysmal positional vertigo (BPPV) is a condition with recurrent attacks in a significant proportion of patients. The present case- control study was conducted to assess the influence of serum vitamin D normalization on recurrent attacks of vitamin D deficient patients. Methods: Diagnosis of BPPV was made based on history and clinical examination and exclusion of other c...
متن کاملNormalization of qPCR array data: a novel method based on procrustes superimposition
MicroRNAs (miRNAs) are short, endogenous non-coding RNAs that function as guide molecules to regulate transcription of their target messenger RNAs. Several methods including low-density qPCR arrays are being increasingly used to profile the expression of these molecules in a variety of different biological conditions. Reliable analysis of expression profiles demands removal of technical variati...
متن کاملGroup Normalization for Genomic Data
Data normalization is a crucial preliminary step in analyzing genomic datasets. The goal of normalization is to remove global variation to make readings across different experiments comparable. In addition, most genomic loci have non-uniform sensitivity to any given assay because of variation in local sequence properties. In microarray experiments, this non-uniform sensitivity is due to differe...
متن کاملData-driven intensity normalization in PET group comparisons
Short Title: Data-driven intensity normalization in PET group comparisons ABSTRACT Background: Global mean (GM) normalization is one of the most commonly used
متن کامل